lazytables: Faster distributed machine learning through staleness
نویسنده
چکیده
Actifio American Power Corporation EMC Corporation Emulex Facebook Fusion-io Google Hewlett-Packard Labs Hitachi, Ltd. Huawei Technologies Co. Intel Corporation Microsoft Research NEC Laboratories NetApp, Inc. Oracle Corporation Panasas Samsung Information Systems America Seagate Technology STEC, Inc. Symantec Corporation VMware, Inc. Western Digital LazyTables .................................... 1
منابع مشابه
Trading Freshness for Performance in Distributed Systems
Many data management systems are faced with a constant, high-throughput stream of updates. In some cases, these updates are generated externally: a data warehouse system must ingest a stream of external events and update its state. In other cases, they are generated by the application itself: large-scale machine learning frameworks maintain a global shared state, which is used to store the para...
متن کاملGlobal Warming: New Frontier of Research Deep Learning- Age of Distributed Green Smart Microgrid
The exponential increase in carbon-dioxide resulting Global Warming would make the planet earth to become inhabitable in many parts of the world with ensuing mass starvation. The rise of digital technology all over the world fundamentally have changed the lives of humans. The emerging technology of the Internet of Things, IoT, machine learning, data mining, biotechnology, biometric, and deep le...
متن کاملFaster Asynchronous SGD
Asynchronous distributed stochastic gradient descent methods have trouble converging because of stale gradients. A gradient update sent to a parameter server by a client is stale if the parameters used to calculate that gradient have since been updated on the server. Approaches have been proposed to circumvent this problem that quantify staleness in terms of the number of elapsed updates. In th...
متن کاملExploiting Bounded Staleness to Speed Up Big Data Analytics
Many modern machine learning (ML) algorithms are iterative, converging on a final solution via many iterations over the input data. This paper explores approaches to exploiting these algorithms’ convergent nature to improve performance, by allowing parallel and distributed threads to use loose consistency models for shared algorithm state. Specifically, we focus on bounded staleness, in which e...
متن کاملStaleness-Aware Async-SGD for Distributed Deep Learning
This paper investigates the effect of stale (delayed) gradient updates within the context of asynchronous stochastic gradient descent (Async-SGD) optimization for distributed training of deep neural networks. We demonstrate that our implementation of Async-SGD on a HPC cluster can achieve a tight bound on the gradient staleness while providing near-linear speedup. We propose a variant of the SG...
متن کامل